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ABSTRACT 

Dissertation quality may be improved by use o£ student peer 
review as a quality control mechanism. An actual data set 
Involving ratings o£ 52 dissertations is employed to illustrate 
possible design and analytic choices that may be helpful in using 
such a model. Strategies for evaluating rating consistency and 
validity are illustrated. Analyses to address four types of 
program policy analysis questions are also illustrated. 
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Perhaps the major challenge confronting doctoral prograr^s is 
maintaining high standards of quality while also respecting 
student rights and prerogatives. For example, NCATE accreditation 
regulations require that students must have a significant voice 
in the assignment of advisors and committee members. Yet some 
students may be prone to select committee members for t leir 
flexibility rather than for their expertise. 

The quality of dissertations must remain a fundamental 
concern of doctoral faculty, however. Even if a faculty member ^s 
own inherent interest in scholarship is not sufficient to warrant 
interest in the quality of dissertations being produced under the 
faculty member's own direction or under the direction of 
colleagues, interests in the survival of the program may itself 
warrant concern. 

External reviewers who make recommendations to state boards 
regarding program expansion or termination tend to pay particular 
attention to the quality of dissertations produced by doctoral 
programs. For example, in a recent review of education doctoral 
programs in Louisiana, the review team spent the preponderance of 
its time during the review reading dissertations. The team not^d 
that the quality of dissertations varied markedly and that, in 
particular, studies often were not adequately grounded in theory. 
In any case, in its report to the Louisiana Board of Regents the 
review team seemed to base its major recommendations upon the 
quality issue: 

The program at UNO [University of New Orleandsl 
deserves more support than it has been receiving. 
In the judgment o£ the team, UNO has done the most 



of any institution in the State with what they 
have. They have certainly been the most efficient. 
They produce the best dissertations and the single 
best doctoral program, namely. Special 
Education... The faculty is underpaid and the team 
wonders how long they will stay. It is strongly 
recommended that UNO's doctoral program in 
education be allowed the same level of resources 
as obtained at LSU. (p. 87) 
Various models for maintaining dissertation quality have 
been developed. For example, some programs require that advisors 
must be selected from a pool of persons with widely acknowledged 
expertise as researchers, other programs require the Dean of 
Graduate Studies to supervise every committee. Some programs vest 
responsibility in the Dean of Graduate studies for personally 
certifying that each dissertation meets accepted standards. 

An additional model might be used to compliment any of these 
various other models. Current students themselves might be 
routinely required to evaluate the dissertations produced by 
recent cohorts of previous students. The theory of such a model 
is that even less scholarly doctoral students might be more 
diligent in their attention to quality if they anticipated future 
systematic reviews of their work. The review might be an 
anticipatory check for at least some students who may be 
sensitive to embarrassment; the process of serving as a reviewer 
advises current students that their works will be closely 
scrutinized. Futhermore, doctoral faculty might be stimulated to 
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Impose higher standards of quality if they felt that they might 
be identified with studies that were either systematically best 
or systematically worst. 

The problem with such an endeavor Is that numerous studies 
must be evaluated to provide reasonable comparisons, and numerous 
raters must be employed so that confidence can be vested In 
rendered judgments. New cohorts of doctoral students provide a 
natural pool of reviewers that can be used for these purposes. 

The purpose of the present paper Is to Illustrate some 
design and analytic choices that might be employed in such a 
model. Actual data from a small sample (n=9) of ratings are 
employed to Illustrate possibilities. All education dissertations 
(52) produced during a five year period at one university were 
rated. A list of dissertation authors by program area Is 
presented in Appendix A. 

Design Issues 

Typically it will not be feasible to require all raters to 
read all dissertations. Matrix sampling techniques can be 
employed to make the rating task manageable. However, a core of 
dissertations that will be rated by all subjects can be 
identified so that interrater reliability can be investigated. 
Generally, the core should be selected to represent the full 
range of dissertations with respect to quality; this allows the 
ratings to be more variable and thus estimates of interrater 
reliability become more accurate, m the present example, 14 
dissertations were rated by all raters and the remaining 
dissertations were typically rated by two or three raters. 
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It la important to evaluate the consistency and the validity 
of the ratings. External ratings data can be employed to 
Investigate concurrent validity. For example, m the present 
example all dissertations had been reviewed by a doctoral faculty 
committee for the purpose of designating annual winners and 
finalists of an outstanding dissertation award. The two 
variables, finalists and winners, were dummy coded and correlated 
with each reviewers ratings. These correlation coefficients are 
reported In Table 1; the coefficients were computed using 
palrwlse deletion of missing cases in the calculation o£ each 
coefficient so that each coefficient was based on all available 
ratings data. 

Table 1 

Rater Consistency and Validity 
123456789 Fin 

Rater 1 

2 .46 

3 .40 .66 

4 .56 .74 .87 

5 .71 .79 .84 .89 

6 .57 .63 .69 .84 .86 

7 .71 .82 .85 .85 .92 .76 

8 .74 .85 .55 .85 .91 .81 .92 

9 .64 .68 .68 .72 .75 .81 .77 .84 
Finalist .45 .37 .41 .51 .57 .63 .44 .52 .57 
Winner .55 .14 .21 .12 .45 .32 .14 .29 .39 .60 

Q-technlque factor analysis provides a vehicle for 
evaluating the Interrater consistency and validity of the 
judgments. The method groups "raters" into clusters based upon 
the similarities of their judgments. All factors with eigenvalues 
greater than one were extracted. The eigenvalue of the first 
factor (6.34) Indicated that it accounted for 57.6% of the 
variance in the correlation matrlc. Factor ii accounted for an 
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additional 13.7% percent o£ the variance In the matrix. Figure i 

presents the location of the raters In the factor space defined 

by the varlmax rotation of the principal components analysis of 
the Table 1 matrix. 



Figure 1 
Raters Arrayed In Factor Space 
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The second factor appears to be a measurement artifact 
arising from the analysis of two dummy coded variables consisting 
of ones and zeros together with analysis of Intervally-scaled 
ratings provided by the raters. Overall, Inspection of the Table 
1 correlation coefficients and Interpretation of the Q-technlque 
eigenvalue for Factor I Indicates that the raters judgments v/ere 
reasonably consistent and valid. 

Substantive Interpretations of Ratines 
In addition to serving as an educative process, the proposed 
model can also be used for formative program evaluation purposes. 
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or to address policy questions in service of decision-making. 
Some selected examples of some analyses may illustrate the 
possibilities. The analyses use forms of the ratings data, such 
as the data presented in Table 2. The table presents the mean 
rating of each dissertation, along with the standard deviation of 
tho ratings of each dissertation. 

A rather elementary but important question to ask considers 
the issue of overall quality of the dissertations at a school and 
the homogeniety of quality across studies. In the present example 
the mean of the mean ratings of the 52 studies was 11.1 (SD=4.2>. 
The studies were rated on 19- interval scales to maximize 
reliability of the ratings, and 19 represented the most favorable 
rating. The dissertation ratings ranged from 2.89 for one study 
to 17.11 for the most favorably rated dissertation. The 
coefficient of skewness (-.25) indicated that ratings were 
symmetrically distributed about the mean rating. These results 
suggest that dissertation quality was perceived as being highly 
variable. 

Another policy question of potential interest considers 
possible systematic variations in dissertation quality across 
program areas. As noted in Appendix A, four program areas were 
represented in the example data set. Table 3 presents mean 
ratings for each program area. The results indicate some 
tendency toward systeioatic differences in quality across 
programs. However, the standard deviation for Program 1 suggests 
that variability in dissertation quality may be particularly 
troublesome with respect to this «1egree program. 



ERIC 



6 

9 



Descriptive Statistics for Rated Dlssertatl 

Mean SD 

2.889 2.759 

3.000 2.784 

3.500 2.121 

4.500 2.121 

5.000 .000 

5.000 1.414 

5.500 .707 

6.000 4.243 

6.000 2.828 

6.667 5.686 

7.500 6.364 

8.000 8.485 

8.000 .000 

8.333 4.509 

8.333 4.933 

8.500 6.364 

9.000 6.245 

9.000 .000 

9-222 5.696 

9.333 5.5C8 

9.500 6.364 

9.667 4.726 
10.000 5.657 
10.000 5.657 
10.667 5.033 
11.000 1.732 
11.500 2.121 
11.667 5.686 
11.667 4.933 
12.500 4.950 
12.500 4.950 
13.000 1.414 
13.500 .707 
13.500 2.121 
13.667 3.873 
14.000 2.828 
14.000 3.000 
14.333 2.872 
14.778 2.539 
14.889 3.551 
15.333 3.279 
15.500 3.536 
16.000 3.240 
16.222 3.270 
16.222 2.539 
16.333 4.743 
16.333 2.915 
16.500 2.536 
16.500 2.121 
17.000 1.414 
17.000 .000 
17.111 1.167 
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Table 3 

Mean Ratings Across Programs 



Mean SD 

Program 1 9.82 6.20 

2 9.89 3.79 

3 12.33 3.85 

4 13.39 3.36 



A third issue of possible interest considers whether 
dissertation quality is perceived as improving over time. 
Fourteen semesters were represented within the pool of rated 
dissertations in the present example. Table 4 reports descriptive 
statistics for ratings of dissertations conducted during each of 
these semesters. Polynomial trend ANOVA identified no noteworthy 
trends in the Table 4 means. 

Table 4 

Descriptive Statistics over Time 



Time 


Mean 


SD 


1 


10.8 


5.7 


2 


8.4 


2.4 


3 


12.4 


3.5 


4 


10.4 


6.3 


5 


12.7 


4.5 


$ 


10.9 


3.1 


7 


11.7 


0.0 


8 


7.7 


6.0 


9 


9.6 


7.3 


10 


10.5 


7.1 


11 


12.8 


1.8 


12 


11.7 


0.0 


13 


12.5 


5.6 


14 


10.7 


3.2 



A fourth analysis considers whether particular major 
professors are more likely to be associated with dissertations 
that are perceived as being of higher quality. Twenty faculty had 
served as major professors for at least one of the dissertations 
in the rated pool. The mean ratings for studies associated with 
given faculty ranged from a low of 4.75 (SD=1.77) for one faculty 
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member to a high of 17.00 (SD=.00> for a faculty member who only 
served as major professor one time during the five year period 
under consideration. Results from a one-way analysis of variance 
of these differences is presented in Table 5. These results 
suggest that the programs may not have been wisely utilizing 
their faculty during the time period considered in the study. 

Table 5 
One-way ANOVA 

E £ f 6 c t 

g%„.e„o. fys II f„.„ I!?-. 



** 



E. < .01 



Summary 

Dissertation quality may be improved by use of peer review 
as a quality control mechanism. A small data set involving 
ratings of 52 dissertations was employed to illustrate possible 
design and analytic choices that may be helpful in using such a 
model. Strategies for evaluating rating consistency and validity 
were illustrated. Analyses to address four types of program 
policy analysis questions were also illustrated. 
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Appendix a: Rated Dissertations 



Name \Deg 

C FAUST BdD 

A LANE PhD 

0 LEVITOV PhD 

A STEEPLEY EdD 

C ANDBRSSON PhD 

S ARAKBURO BdD 

S CARLTON EdD 

A DUMAS BdD 

C ORBENE,J BdD 

0 JEANSONNE PhD 

A KAPPELMAN EdD 

0 LUMPKIN PhD 

C MCCULLOCH PhD 

A MELANCON BdD 

A MEZA BdD 

A PHILLIPS BdD 

A RICHARDSON PhD 

S SLATE BdD 

A TAYLOR PhD 

A BBCNBL EdD 

A DUCOTE PhD 

A FBOLA PhD 

C GAUDIN EdD 

A MILLER,«M EdD 

C PITTS PhD 

A OUAIN PhD 

S SABRIO PhD 

A SCAFIDEL EdD 

A TKERIOT BdD 

A ZEI6LER PhD 

C BORRELLO EdD 

A BROOKS PhD 

A HARRIS BdD 

S JACKSON EdD 

S KBLLBY PhD 

C SULLIVAN BdD 

O TBXIDOR PhD 

O THOMAS PhD 

S BANBURY PhD 

A BARTON PhD 

C BEALL PhD 

C CASBERGUE PhD 

O G ARLINGTON PhD 

S MELVILLE EdD 

8 MILLER, JAY PhD 

A WILLIAMS PhD 

A ALEXIS PhD 

A BUSENBARK PhD 

A CARTER PhD 

A GRBEN,ROS PhD 

S KRBIG PhD 

S SISTBRHEN PhD 



bis Abst 
41/4056A 
4 174 9 13 A 
42/0143A 
42/0046A 
42/2993A 
42/2604A 
42/2606A 
42/1872A 
42/4816A 
4 2/250 2 A 
42/2405A 
42/2893A 
42/2528A 
42/1882A 
42/1882A 
42/4676A 
42/2421A 
42/2628A 
42/2061A 
43/2168A 
43/1888A 
43/2842A 
43/1914A 
43/2189A 
43/1910A 



U Microfilms 
81-12,898 
81-12,899 
81-13,580 
81-13,581 
81-29,909 
81-25,880 
81-25,881 

81- 23,012 

82- 08,060 
81-25,882 
81-25,884 
81-26,222 
81-25,885 
81-23,013 

81- 23,017 

82- 08,010 
81-25,886 
81-25,887 

81- 23,015 

82- 27,568 
82-23,262 
82-23,263 
82-25,264 
82-27,570 
82-23,261 



43/1932A 82 
43/2850A 82 
43/3828A 83 
43/2197A 82 
45/0065A 84 
44/2301A 83- 
45/0094A 84 
45/0147A 84- 
44/2717B 83- 
44/2425A 83- 
44/2369A 83- 
44/2682A 83- 
45/2062A 84- 
45/2316A 84- 
46/2186A 85- 
46/OllOA 85- 
4S/1342A 84- 
46/0123A 85- 
46/0128A 85- 
46/0044A 85- 
46/2200A 85- 
46/1451A 85- 
46/1527A 85- 
46/1458A 85- 
46/1902A 85- 
46/1597A 85- 



-23,267 
-23,268 
-10,891 
-27,571 
-08,136 
-28,108 
■08,137 
-09,292 
-28,209 
-28,214 
-28,215 
-28,216 
•17,612 
-25,266 
•07,561 
•07,562 
•17,613 
•07,563 
07,564 
•07,566 
16,301 
16,302 
16,303 
16,304 
16,305 
16,307 



Program Area Codes 
*'A"«Administration 
"C"«Currlculum 
"O^sCounseling 
. "S"=Special Educatic 
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